Housing market experts are keeping a close eye on the Federal Reserve as they anxiously await word on whether the agency will start pulling back on its controversial stimulus program, known as quantitative easing.

Since September of last year, the Fed has been buying $85 billion in mortgage-backed securities and Treasury bonds a month to help support the economy. The purchases have been credited for the historically low mortgage rates seen this year, which ultimately helped stimulate home sales and boost prices.

Related: How much will your monthly mortgage payment be? Find out

But now the Fed is expected to announce that it will scale back on its bond-buying program -- a move that is expected to cause rates to slowly rise, said Doug Duncan, chief economist for Fannie Mae.

The mortgage market has already factored in a modest cutback in the Fed's purchases. Mortgage rates have risen 1.2 percentage points since May when Fed chairman Ben Bernanke mentioned the possibility of reducing the agency's bond-buying program. In June, he noted that the tapering could begin as early as September, if the economic recovery continued on course.

However, even if the Fed started cutting back on its bond purchases this month, many don't expect the cuts to be sizable. "The recovery has been weaker the past couple of months than what the Fed had been talking about," said Duncan. "It would be a surprise if they act aggressively."

Duncan said initial cuts to the bond-buying program likely won't exceed $10 billion a month and most of those cuts will be in Treasuries.

Those expectations were reinforced in a paper presented by Northwestern University economists Arvind Krishnamurthy and Annette Vissing-Jorgensen at an annual meeting for central bankers in Jackson Hole, Wyo., late last month. The economists found that the purchase of mortgage-backed securities served as a more effective stimulus than the purchase of Treasury bonds.

Related: The taper is coming. Get over it

Should the Fed continue to purchase mortgage-backed securities at current rates, there is a chance mortgage rates could actually head lower, said Keith Gumbinger of HSH.com, a mortgage information provider.

Frank Nothaft, chief economist for Freddie Mac, expects the Fed to act slightly more aggressively. He anticipates that the agency will cut its purchases of Treasuries by about $10 billion a month and reduce its purchases of mortgage-backed securities by $5 billion. He expects the Fed to continue to scale back its purchases through mid-2014, when the program would end entirely.

As a result, Nothaft expects mortgage rates to climb to close to 5% by next June. Last week, the 30-year fixed rate averaged 4.6%.

Quiz: How much do you know about mortgages?

Despite higher mortgage rates, only a small number of potential homebuyers -- those who would have to struggle to afford a home in the first place -- would put the brakes on their purchases, said Nothaft.

Remaining buyers would probably welcome a slight cooling off in the housing market. Home prices were up 12.1% in June compared with a year earlier, according to the S&P/Case-Shiller home price index.

That kind of surge has stirred fears of a new housing bubble and many economists -- and homebuyers -- wouldn't mind seeing price gains come back to more normal levels.